1. Field of the Invention
The present invention relates to a technique for calculating the position and orientation of an index present on a physical space.
2. Description of the Related Art
As one of important techniques for providing high-quality mixed reality, a registration technique of a physical space and virtual space is known. As the registration technique, various conventional methods have been proposed. Of these methods, a method of calculating (calibrating) the position and orientation of a camera from an image (index image) obtained by capturing an image of an index is popularly used in terms of cost and convenience.
Most of these methods are premised on that the positions and orientations of all indices on the space are given, and require measurement and input of the positions and orientations of the indices, in advance. As the indices, artificial markers (to be simply referred to as markers hereinafter) or natural features are generally used. When the markers are used, it becomes impractical to manually measure the positions and orientations of the markers as the number of markers to be allocated increases.
In order to broaden the range of the space that allows the registration processing, a large number of markers must be set over a broad range. Therefore, means for efficiently and precisely measuring the positions and orientations of these large number of markers set over a broad range is required.
In recent years, a method of calibrating the relative positions and orientations of markers based on an image obtained by capturing an image of markers has been proposed (see D. Kotake, S. Uchiyama, and H. Yamamoto, “A marker calibration method utilizing a priori knowledge on marker arrangement,” Proc. Third IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'04), PP. 89-98, November 2004).
After the relative positions and orientations of all markers by this method, if the position and orientation of an arbitrary marker on a world coordinate system are determined, the positions and orientations of all the markers on the world coordinate system can be calculated.
An overview of the operation required for the registration processing and the calibration processing in this method is as follows.
(1.1) Markers are generated and are allocated on a space that is to undergo registration processing.
(1.2) The position and orientation of a marker (reference marker) whose position and orientation on the world coordinate system are determined are input.
(1.3) An image of the markers is captured.
(1.4) Calibration is executed.
(a) The relative position and orientation between one marker and the camera are calculated based on the image including a plurality of markers.
(b) The relative position and orientation between another marker and the camera are calculated.
(c) The relative position and orientation between the two markers are calculated.
(d) These processes are executed for all images including the plurality of markers, and the relative positions and orientations among all of the markers are calculated.
(e) The absolute positions and orientations of all the markers (relative markers) other than the reference marker are calculated based on the reference marker.
(1.5) The registration processing is executed using the markers whose positions and orientations are calculated by the calibration.
Registration using common natural features includes two processes, i.e., generation of a natural feature database and estimation of the position and orientation of a camera using the natural feature database (Iryna Skrypnyk and David G. Lowe: “Scene Modelling, Recognition and Tracking with Invariant Image Features”, Proc. Third IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 110-119, 2004). The natural feature database is generated using a large number of images obtained by capturing images of the physical space as in the marker calibration. An overview of the operations required for the registration processing using natural features is as follows.
(2.1) A plurality of images of the physical space are captured.
(2.2) Features are extracted from the images.
(2.3) The position of a given feature is calculated from the plurality of images.
(a) Images of that feature on the plurality of images are associated with each other.
(b) The relative position and orientation among the plurality of images are calculated based on the association.
(c) The three-dimensional (3D) position of the feature is calculated based on the relative position and orientation.
(d) The above processes are executed for all of the extracted features.
(2.4) The position and orientation of the camera are estimated using the generated natural feature database to execute registration.
With the existing method, when markers are used, all of the markers captured in the operation (1.3) are calibrated in (1.4) above, and are used in the registration processing (1.5).
As the number of markers used increases, the registration precision improves. In such case, however, since the load on the registration processing and calibration processing becomes heavier, it is desirable to use a smaller number of markers. When the operator carefully captures an image of only required markers, the number of markers can be reduced to some extent. However, such operation requires operator's experience and skill. For this purpose, in practice, the operator repeats the operations (1.1) to (1.5) to empirically determine an optimal allocation and number of markers.
At this time, with the existing method, when an arbitrary marker is excluded from the calibration or registration processing, all or some of the captured images are discarded so that the marker to be excluded is not included in the captured images. After that, the calibration is executed again. Or after the marker is removed from the space, images are captured again to execute the calibration.
When a plurality of image processing apparatuses which perform calibration and registration processing are used in neighboring spaces, a given image processing apparatus is more likely to accidentally capture an image of markers which are set to be used by the neighboring image processing apparatus in addition to those to be used by itself in the operation (1.1). In this case, unnecessary registration processing and calibration processing must be done.
Furthermore, the marker IDs of the markers which are set to be used by the self image processing apparatus are more likely to overlap those of the markers which are set to be used by the neighboring image processing apparatus. In this case, the calibration and registration processing do not work correctly.
Such marker ID overlapping also occurs when markers which are generated as auxiliary markers and are carelessly left on the space where the calibration and registration processing are done are accidentally captured. Also, the marker ID overlapping occurs by recognition errors of markers in the image processing.
These recognition errors of markers occur due to disturbances such as image capturing an object similar to a marker, parameters such as an exposure value, stop value, and the like of a device for capturing an image of markers, a light condition in an image capture environment, and the like, and it is difficult to avoid such recognition errors even when the operator of the image processing apparatus exercises extra caution.
With the existing method, in order to exclude overlapping markers from the calibration or registration processing, after all or some of captured images are discarded, an image of all markers except for the overlapping markers is captured again.
On the other hand, upon using natural features, the positions of features extracted in the operation (2.2) are calculated by the processing (2.3) to generate a natural feature database, which is used in the registration processing (2.4).
With the existing method, many similar features may appear. In this case, upon generation of the natural feature database or execution of the registration processing, similar features may be associated with each other as identical features, thus causing operation errors.